Evaluating Modes of Teacher Preparation: 

A Comparison of Face-to-Face and Remote Observations of Graduate Interns 


Abstract 

Although technology-mediated learn¬ 
ing has its advantages , it does not 
come without operational tradeoffs. 
Thus, consideration of the effective¬ 
ness of new modes of teacher prepa¬ 
ration in comparison to traditional 
longstanding , research-proven meth¬ 
ods is necessary. In this study we 
evaluated one large southern urban 
universitys implementation of syn¬ 
chronous remote observations of 
teacher education candidates using 
live classroom software. We compared 
remote and face-to-face observations 
to determine if the modes of observa¬ 
tion were equivalent in supporting 
professional growth and evaluating 
quality instruction. Results suggest 
that both modes of observation are 
not equal but are comparable in sup¬ 
porting graduate interns professional 
growth and in measuring teaching 
effectiveness. The implications of this 
study shed light on the advantages 
and limitations of emergent technolo¬ 
gies as teacher education programs 
explore alternative forms of Web- 
based evaluations of teacher candi¬ 
dates. (Keywords: Teacher education , 
student teaching, internship , remote 
observation , video) 


I mpending changes within teacher 
education, whether motivated by 
national or state educational man¬ 
dates, teacher shortages, shifting teacher 
education candidate needs, growth of 
second-career professionals seeking em¬ 
ployment, or tighter operating budgets, 
have paved new and innovative avenues 
for teacher preparation programs. These 
changes come at a time when emerg¬ 
ing technologies are being explored 
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as pedagogical tools for new learning 
pathways. Although many education 
programs have investigated or imple¬ 
mented online coursework, the chal¬ 
lenges of conducting clinical experiences 
and teaching observations in a virtual 
setting still pose a barrier that limits the 
scope of online licensure opportunities. 
Adoption of cutting-edge processes does 
not necessarily translate into positive 
learning outcomes, as gaps in educa¬ 
tional research linking the two abound 
(U.S. Department of Education, 2009). 
Although technology-mediated learn¬ 
ing has its advantages, it does not come 
without operational tradeoffs. Thus, 
consideration of the effectiveness of 
these new modes of teacher preparation 
compared to traditional, longstanding, 
research-proven methods is necessary. 

In this study, we evaluated one large 
southern urban university’s implementa¬ 
tion of synchronous remote observations 
of teacher education candidates using live 
classroom software. We compared remote 
and face-to-face observations to deter¬ 
mine if the modes of observation were 
equivalent in supporting professional 
growth and evaluating quality instruc¬ 
tion. Results suggest that both modes of 
observation are not equal but are com¬ 
parable in supporting graduate interns’ 
professional growth and in measuring 
teaching effectiveness. The implications 
of this study shed light on the advantages 
and limitations of emergent technologies 
as teacher education programs explore 
alternative forms of Web-based evalua¬ 
tions of teacher candidates. 

Overview of the Research Project 

With the push to address teacher 
shortages and increasing population 
growth, universities, colleges, and state 


departments across the nation have 
sought alternative processes for licens¬ 
ing teachers. The rise of urban-based 
programs such as Teach for America, re¬ 
gional alternative licensing centers, and 
postbaccalaureate licensing programs 
are evidence of innovative strategies de¬ 
signed to attract and recruit second-ca¬ 
reer professionals to teaching. Although 
these programs sought to address the 
needs of urban centers, they fell short of 
meeting the needs of more remote and 
rural areas outside of the geographic 
boundaries of the facilitating institution. 
Simultaneously addressing the demands 
of schools not within the proximity of 
licensing organizations and interests 
of geographically bound professionals 
necessitates rethinking instructional 
practices and re-visioning programmatic 
structures. One college of education at a 
large southern urban university has en¬ 
gaged in addressing these unmet needs 
through the implementation of a 100% 
online graduate teacher licensure and 
master’s program. 

Although educational faculty at this 
particular university willingly accepted 
administrative challenges to expand 
licensure opportunities to address state¬ 
wide teacher shortages, most articu¬ 
lated a belief that technology-mediated 
instruction has limits. Thus, the initial 
design of an online licensure program 
for second-career professions was 
limited to only lateral-entry teachers. 
Lateral-entry teachers are professionals 
who have a content-area degree (such 
as a BA in history) and are temporarily 
employed in public schools as teachers 
in a related field (such as social studies) 
but do not hold a teaching license. Jus¬ 
tification for this decision was twofold: 
First, by national standards (No Child 
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Left Behind), lateral-entry teachers 
are not considered “highly qualified” 
and are required to complete relevant 
pedagogical and content preparation 
for teacher licensure. Second, the state 
department accepted school-based clini¬ 
cal observations of lateral-entry teachers 
as evidence of successful completion 
of the graduate internship, whereas all 
nonteaching licensure candidates must 
be observed by the licensing institution 
during the student teaching experience. 

Although university faculty could 
envision similar processes for online and 
face-to-face instruction in all peda¬ 
gogical coursework, they were troubled 
with the challenge of how to facilitate 
classroom observations during the 
student teaching experience (referred to 
from this point forward as the graduate 
internship). Second-career profession¬ 
als who are not employed as teachers 
are required to complete an internship 
experience similar to student teaching— 
a semester teaching practicum with an 
experienced, practicing teacher. Deep- 
seated beliefs about quality teacher prep¬ 
aration and research-based strategies 
for scaffolding the professional growth 
of novice teachers inhibited educational 
faculty from considering alternatives to 
onsite observations of teacher education 
candidates. Consequently, internship 
observations since the inception of the 
online program had to occur within 
driving distance from the university so 
that observations could be conducted 
face to face; resulting in geographic 
boundaries (the metropolitan region 
and 14 surrounding counties) defining 
the scope of programmatic outreach. 
With continual demands of expanding 
the teacher licensure program to address 
statewide teacher shortages, a growing 
untapped market of potential consum¬ 
ers, and increasing availability and 
cost-effectiveness of emerging technolo¬ 
gies, this urban university administra¬ 
tion shifted its vision to more aggres¬ 
sively adopt innovative platforms for 
instructional delivery. These sentiments, 
exacerbated by economic pressures, 
state-mandated declines in operating ex¬ 
penses, and overall long-term budgetary 
cuts, created an urgency for exploring 


strategies that would reduce costs but 
not sacrifice the quality of instruction, 
giving credence to the re-visioning of 
existing programs that could potentially 
be expanded and reconceptualized 
through technology. In response to these 
compounding forces, faculty within the 
college of education began to reconsider 
how technology tools could facilitate an 
equivalent observational process, over¬ 
coming previous barriers and expanding 
the graduate master’s of arts in teaching 
degree to a 100% online program. At the 
core of this dialogue was the concern 
of whether or not teaching could be 
effectively measured through online 
observations. 

In this study, we explored this issue of 
quality in terms of how technology-me¬ 
diated learning compares to a more tra¬ 
ditional approach to teacher preparation. 
Specifically, we examined the differences 
between two modes of teaching observa¬ 
tions: face-to-face (in-class) observations 
and synchronous remote observations 
of graduate interns. In our compara¬ 
tive study, we evaluated the differences 
between the two observational modes 
and whether these differences affected 
the quality of teacher preparation. The 
implications of our study contribute to a 
growing body of research evaluating the 
quality and effectiveness of technology- 
based teaching and learning. 

Identifying Research and Technology- 
Based Solutions to Teacher Training 

We conducted an initial literature review 
to evaluate existing technology-based 
solutions to teacher preparation and 
to determine their viability in address¬ 
ing our programmatic and stakeholder 
needs. The use of technological applica¬ 
tions that support synchronous teacher 
preparation activities at a distance is 
not new. Videoconferencing tools, for 
example, have been used in teacher 
education programs in a variety of ways 
(Hixon & So, 2009; Northwest Educa¬ 
tional Technology Consortium, 2005). 
These include remote communications 
and observations with students and 
teachers in real classrooms, administra¬ 
tion of field experiences in simulated 
environments, and the facilitation of 


remote supervision and communica¬ 
tion during traditional field experiences 
(Hixon & So, 2009). A great deal of re¬ 
search has been conducted regarding the 
effectiveness of the uses of technological 
applications as time- and cost-effective 
solutions to engaging geographically 
distributed personnel in information 
exchanges related to real-time obser¬ 
vation of praxis. For example, Bonk, 
Malikowski, Angeli, and East (1998), 
Kale, Hur, Yerasimou, and Brush (2006), 
and Moffett (2001) developed video- 
supported online discussions and com¬ 
munities to promote sharing of student 
experiences and diverse interpretations 
of events. Additionally, Vannatta and 
Reinhart (1999) used videoconferencing 
to afford preservice teachers enrolled 
in methods courses opportunities to 
observe and communicate with expert 
teachers integrating technology into 
their classrooms, resulting in a positive 
effect on the technological proficiency 
and integration skills of both university 
faculty and preservice teachers. 

Extending this concept, Johnson, 
Maring, Doty, and Fickle (2006) and 
O’Connor, Good, and Greene (2006) 
used videoconferencing as a tool to 
mentor student teachers as well as to ob¬ 
serve and interact with real classrooms. 
Both of these projects resulted in high 
degrees of participant satisfaction with 
the processes as well as perceived learn¬ 
ing. However, these projects were cost 
prohibitive and did not allow time for 
participant reflection, both of which are 
important components of technology- 
mediated solutions in teacher educa¬ 
tion (Benbunan-Fich, Hiltz, & Harism, 
2005). 

As part of a Preparing Tomorrow’s 
Teachers to use Technology (PT3) 
project, Lehman and Richardson (2007) 
added yet another dimension to the use 
of videoconferencing in teacher educa¬ 
tion. In addition to observing the class¬ 
room and the actions of the students 
and teacher, preservice teachers also 
interacted with the children and teach¬ 
ers and prepared and presented a variety 
of enrichment activities. Although this 
work provided teacher candidates with 
opportunities to explore diversity of 
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instructional practices, examine diverse 
settings, and consider innovative ap¬ 
plications of instructional technologies, 
technical difficulties, time involved in 
implementing the videoconferencing, 
and a perceived lack of authenticity by 
the teacher candidates limited the ef¬ 
fectiveness of the process. 

A second application of videoconfer¬ 
encing tools in teacher education involves 
the administration of field experiences 
in simulated environments or “virtual 
practicums” (Zibit & Gibson, 2005). 
Although this area has limited research, 
Foley and McAllister (2005) found that 
simulated field experiences improved 
the teacher identity of candidates, pro¬ 
vided ample opportunities to connect 
theory and practice, improved reflective 
decision-making, and developed collegial 
dispositions, all resulting in an overall 
feeling of being more prepared and con¬ 
fident to enter the classroom. Although 
this possesses significant pedagogical 
benefits, the technical difficulties, cost as¬ 
sociated with production, perceived lack 
of authenticity, and perception of reduced 
complexity are all issues that need to be 
addressed when using videoconferencing 
as a tool to facilitate “virtual practicums” 
(Hixon & So, 2009). 

In response to the rising costs associ¬ 
ated with observing student teachers, a 
number of researchers have experiment¬ 
ed with the use of videoconferencing 
as a tool for supervision. Additionally, 
with improvements in the cost, usability, 
and functionality of video capture and 
sharing tools, as well as other commu¬ 
nicative applications, such processes are 
becoming much more sustainable than 
in the past. Gruenhagen and colleagues 
(1999) and Venn and colleagues (2001) 
used compressed video as a tool to both 
supervise and interact with student 
teachers who, due to geographic barri¬ 
ers, might not have received an adequate 
or appropriate level of support. Al¬ 
though videoconference proved promis¬ 
ing, drawbacks included lack of personal 
contact, the participants distraction by 
the camera, an increased sense of pres¬ 
sure regarding classroom management 
skills, and technical problems. Garrett 
and Dudt (1998) included the use of 


a videoconferencing unit, a network, 
and a room to observe teachers during 
field experiences. One limitation of this 
project was using one room to observe 
3-14 student teachers. Dymond and col¬ 
leagues (2008) used videoconferencing 
to supervise preservice special education 
students. Although findings suggested 
that the process was potentially a time- 
and cost-effective alternative to the 
supervision of traditional field experi¬ 
ences, technical issues were somewhat 
prohibitive. Problems related to inter¬ 
mittent Internet connectivity, equipment 
setup, sound quality, and visual field all 
reduced the overall effectiveness of the 
process. Lastly, Burrack (2007) used 
videoconferencing to facilitate remote 
observations of student teachers as part 
of a hybrid observation program. Ben¬ 
efits were substantial and included more 
frequent observations, more substantive 
feedback flexibility, and more frequent 
and higher-quality interactions. Cost 
and problems accessing school networks 
due to firewall issues limited the projects 
overall effectiveness. 

Based on the research-based benefits 
of videoconferencing, relatively low-cost 
technologies for implementation and 
continuation, and value of interactive 
communication among all participants, 
we designed a synchronous method of 
observation to parallel the widely ac¬ 
cepted traditional observation method. 
The Remote Observation of Graduate 
Interns (ROGI) was initially supported 
by Centra, a state-of-the-art multime¬ 
dia conferencing platform (Petty & 
Heafner, 2009). We used the designated 
participant tool in Centra’s IP videocon¬ 
ferencing feature during the graduate 
internship to remotely observe teach¬ 
ing, deliver formal observation forms, 
provide immediate feedback on instruc¬ 
tion, and support an interactive dialogue 
among university faculty, geographi¬ 
cally dispersed teacher candidates, and 
cooperating teachers. Although Centra 
was the main software tool, hardware 
was needed to facilitate the two-way 
video and communication exchange. 

We purchased the additional technology 
through a university-funded Academic 
Improvement Grant. Hardware included 


a laptop computer, a camera with video 
streaming capabilities, and a tripod. The 
camera was connected to the laptop. 

The laptop had to be configured with 
each school’s IP address and then con¬ 
nected through the Ethernet cable to the 
school’s server. We repeated this process 
at each school and tested the technol¬ 
ogy prior to the observation to ensure it 
worked properly. A graduate assistant, 
hired through grant funding, traveled 
to the schools and served as videogra- 
pher for all of the observations in the 
pilot semester. We trained the graduate 
intern to set up the technology for each 
observation and to operate the camera 
to capture what the observer directed 
through a text-chat feature in Centra. 

Throughout the piloting of ROGI, 
we considered the following questions, 
which serve as the questions that guided 
our study: 

1. Are remote observations equivalent 

to face-to-face observations? 

2. Are there differences in the two 

processes? 

a. If so, what are these differences? 

b. Do these differences affect the 
quality of teacher preparation? 

Methodology 

During the pilot semester for ROGI, two 
graduate interns located within the same 
school system voluntarily agreed to par¬ 
ticipate in the project evaluation. One 
intern was a lateral-entry teacher cur¬ 
rently employed at a middle school, and 
the other intern was a student teaching 
at a high school. We chose this particu¬ 
lar school system as the pilot location 
for this project for several reasons: (a) its 
willingness to participate, (b) its location 
in regard to the university (the system 
was far enough from the university to 
warrant remote observation but close 
enough to troubleshoot the technology 
when the need arose), (c) its techni¬ 
cal support, and (d) the consent of the 
graduate interns who had been placed 
in this system prior to receiving an 
invitation to participate in ROGI. We 
collected data from graduate intern 
teaching portfolios and postconference 
meetings. A graduate assitant conducted 
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an interview with each graduate intern 
following the internship experience, 
asking the graduate interns to provide 
thoughtful answers to 26 open-ended 
questions. The questions related to their 
experiences with ROGI, their likes and 
dislikes of ROGI, their experiences with 
face-to-face observations, their likes and 
dislikes of face-to-face observations, and 
their preferred mode of observation. The 
graduate assistant recorded, coded, and 
transcribed the interviews. 

Experienced university supervisors, 
content methodologists, and site-based 
personnel conducted the observations 
of graduate interns. The two univer¬ 
sity supervisors, who were also content 
pedagogy experts, observed each gradu¬ 
ate intern a total of five times, and the 
cooperating teacher/school administra¬ 
tors each conducted four observations. 
The cooperating teacher and admin¬ 
istrator s observations all were face to 
face. Two of the university supervisors 
observations were traditional face-to-face 
observations, whereas three were remote 
observations. For the first observation, 
both supervisors observed the intern 
face to face. During the second observa¬ 
tion, one supervisor observed face to 
face, and the second supervisor observed 
remotely. The roles of the supervisors 
were reversed for the third observation. 
Both supervisors observed remotely for 
the final two observations. The college of 
educations director of field experiences, a 
veteran observer with more than 25 years 
experience, participated remotely in the 
fourth observation as a means to trian¬ 
gulate data. We used observer feedback 
and observational data provided on the 
Observation Feedback Form to capture 
various aspects of both processes and 
evaluate the project. We made compari¬ 
sons between observers and by the mode 
of observation, in addition to analyses of 
each graduate intern using observer data 
collected from a required collegewide 
common scoring rubric and open-ended 
feedback form that asks the supervisor to 
identify the strengths of the intern, areas 
of improvement for the intern, and any 
comments or suggestions. 

University supervisors also provided 
journals covering their experiences and 


written reflections on lessons learned. 
These data included the university 
supervisors’ views on the overall experi¬ 
ence, the use of technology, and the 
comparison of observational methods. 

In addition, we evaluated data from 
archived Centra teaching observation 
transcripts for remote observations. 
Centra has a text-chat feature that uni¬ 
versity supervisors used to communicate 
with each other and with the camera 
operator. Finally, the graduate assis¬ 
tant conducted, recorded, transcribed, 
and coded interviews with university 
supervisors following the internship 
experience. 

Each of the researchers read the 
qualitative data, including observational 
transcripts from Centra, supervisor 
reflections, and graduate intern and 
university supervisor interview tran¬ 
scripts, three times. Each researcher 
highlighted commonalities in the 
transcripts and then compared them for 
themes using the constant comparative 
method (Glaser & Strauss, 1967). We at¬ 
tended four meetings to discuss findings 
and interpret emergent themes. Each 
time, we read and reread data to affirm 
researcher agreement on themes and to 
explore participant meaning. After we 
identified patterns and sorted the data 
into domains (Huberman & Miles, 2002; 
LeCompte & Schensul, 1999), we identi¬ 
fied and cited specific examples from 
the narratives to support each of the 
emergent themes. 

We analyzed quantitative data from 
observational tools using a multistep 
process. First, we made statistical 
comparisons using the observation as¬ 
sessment rubric to evaluate differences 
among observers (cooperating teacher 
or school administrator and university 
supervisors) and differences between 
graduate interns (Teacher 1 and Teacher 
2). Second, repeated measures were 
performed by the teacher and between 
observers to measure reliability of data. 
Third, we calculated summative scores 
for each of the 10 variables identified 
within the scoring rubric. Variables 
align with the 10 Interstate New Teacher 
Assessment and Support Consortium 
(INTASC) standards for initial licensure 


(see http://www.ccsso.org/Projects/in- 
terstate_new_teacher_assessment_and_ 
support_consortium/). Finally, we made 
intraclass comparisons to determine if 
differences in observation scores were 
attributed to variance among observers 
or differences between graduate interns. 

We evaluated observational feedback 
data from the open-ended observer 
form using the constant comparative 
method (Glaser & Strauss, 1967) to 
identify emergent themes as recognized 
characteristics of good teaching. We 
met four times to discuss themes and 
confirm consistent meaning of charac¬ 
teristics across observers. Next, we inde¬ 
pendently reread observation data using 
themes to quantify frequency of occur¬ 
rences and evaluation of good teaching 
characteristics. We categorized observer 
evaluations as interns’ strengths or 
weaknesses to allow for comparisons 
across interns. We discussed the data 
and created a common chart measuring 
consistency across observers by intern. 
Qualitative methods of evaluation fol¬ 
lowed established evaluative processes 
(see Marshall & Rossman, 1999; Patton, 
1990). 

Results and Implications 

Are Remote Observations Equivalent 
to Face-to-Face Observations? 

The first research question explored the 
issue of whether or not remote obser¬ 
vations are equivalent to face-to-face 
observations. We define equivalent as 
being the same in effect, value, and 
meaning. We evaluated quantitative 
data to understand if the observational 
processes were interchangeable in al¬ 
lowing consistency in assessing teaching 
performance and in observing attri¬ 
butes of successful teaching behaviors. 
Data from the standardized observa¬ 
tion rubric documented consistency in 
observed teaching improvement over 
four observations for each graduate 
intern. Over time, average observation 
scores improved (see Figures 1 and 2, p. 
158), although the lateral-entry teacher 
(Teacher 1) received higher ratings by 
all observers than the student teacher 
(Teacher 2). Analyses of estimated mar- 
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Figure 1. Average summative evaluation of teaching by observer over time for Teacher 1. 



Figure 2. Average summative evaluation of teaching by observer over time for Teacher 2. 
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Figure 3. Estimated marginal means of observation assessment rubric by teacher over time. 


ginal means for all 10 INTASC standards 
(or measured variables) for initial licen¬ 
sure affirmed reliability of these findings, 
indicating that over time both graduate 
interns demonstrated growth in their 
teaching (see Figure 3). Data support 
an accepted expectation of an effective 
internship program—that over time and 
with support of the university supervisor 
and onsite mentor, interns will improve 
their content and pedagogical knowl¬ 
edge and skills. 

We attributed the observed differences 
in teaching effectiveness as measured by 
the common teaching rubric to the differ¬ 
ences in experiences of the two graduate 
interns and not to the mode of observa¬ 
tion or observer. As presented in Figures 

1 and 2, data indicate consistent agree¬ 
ment across observers over time for each 
teacher. Teacher 1 was a certified elemen¬ 
tary teacher with 6 years of elementary 
teaching experience prior to becoming a 
lateral-entry middle school math teacher. 
Teacher 2 was a second-career profes¬ 
sional student teaching in secondary 
social studies. He had retired from the 
military and chose to student teach rather 
than pursue a lateral-entry position for 
the benefits of having a cooperating 
teacher and supportive internship experi¬ 
ence. Teacher 1 was consistently rated 
higher on all observations and was noted 
to be a more experienced instructor and 
fluid at adapting content and instruction 
to meet diverse student needs. 

Furthermore, to evaluate whether 
or not the mode of observation could 
have affected observer evaluations 
of intern performance, we calculated 
estimated measures of marginal means 
for each of the 10 variables measured 
within the standardized assessment 
rubric by observer and over time for 
each graduate intern. The data that 
Figures 4 and 5 present indicate agree¬ 
ment across observers and over time 
for all 10 variables. Observer 1 was the 
school administrator, and Observers 

2 and 3 were the university supervi¬ 
sors. For example, Teacher 1, although 
an experienced teacher, initially did 
not provide the level of depth in her 
instructional plans needed for success. 
Although her planning (Variable 7), 
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Figure 5. Estimated marginal means by observer for Teacher 2’s student motivation and classroom management. 


as evaluated by university supervisors 
and administration, was acceptable 
(Level 3) at the first observation, she 
did not demonstrate exemplary (Level 

4) pedagogical thinking until her third 
observation. Data indicate agree¬ 
ment across observers even though 
observations occurred both face to 
face and remotely. Likewise, Teacher 

2 struggled with student motivation 
and classroom management (Variable 

5) in the first observation (Level 2 or 
below expectations) but made gradual 
improvements in his skills over the 
next three observations (as noted by 
observer evaluations of levels of up¬ 
per 2s to 3s). The pattern in observer 
evaluations of teaching behaviors was 
consistent for each observation wheth¬ 
er or not the observation occurred face 
to face or remotely. 

With opportunities to practice man¬ 
agement skills and pedagogy and build 
content expertise, the accepted expecta¬ 
tion among teacher educators are that 
interns should demonstrate growth over 
time. Data from the observation assess¬ 
ment rubric and data from open-ended 
evaluation forms support this general 
assumption. Aforementioned quantita¬ 
tive data document intern growth over 
time (see Figures 1-5). 

Evaluation of open-ended data 
presented a richer understanding of 
the observed characteristics of good 
teaching behaviors. Analyses of quali¬ 
tative data revealed emergent themes 
of observed characteristics of teach¬ 
ing: instructional strategies, classroom 
management, organization, question¬ 
ing techniques, student engagement, 
content, feedback, and assessment. To 
determine reliability and consistency in 
feedback across observers participating 
in a synchronous (remote) observation, 
we noted the frequency of emergent 
themes and whether these were identi¬ 
fied as strengths or weaknesses. For the 
purpose of this study, we selected the 
fourth observation for each graduate 
intern to qualitatively evaluate inter¬ 
rater reliability in a remote observation. 
The purpose of using this observation 
was to examine if observers were able to 
consistently measure common traits of 


good teaching in an online setting. As an 
additional attempt to cross-validate data, 
the fourth observation included com¬ 
parisons with an extensively experienced 
observer, the director of field experi¬ 
ences. All three observers conducted this 
observation remotely and independent 
of other observers. Tables 1 and 2 (p. 

160) present the findings. 

Overall, observers generally agreed 
in all observation categories for both 
graduate interns. Observers were 
consistent in identifying common ele¬ 
ments of effective teaching. They also 
noted similar areas for improvement. 
There was consensus among observers 
that Teacher 1 was a stronger teacher 
and exhibited successful pedagogical 
strategies. Teacher 2 had more areas 
for improvement across observers. 

There was some variance in frequency 
of noted strengths and improvements, 


although overall agreement exists. Dif¬ 
ferences can be attributed to observer 
expertise. University Supervisor A is a 
math methods specialist, holds initial 
and advanced licensure in middle and 
secondary mathematics, and was the 
lead instructor for Teacher 1. University 
Supervisor B was the lead instructor 
for Teacher 2, holds advanced licensure 
in grades 6-12 and secondary social 
studies and initial licensure in grade 
6-12 mathematics, and is the social 
studies methods specialist. The direc¬ 
tor of field experiences has 25 years 
of observational experience and holds 
advanced licensure in all core content 
areas, including mathematics and social 
studies. Overall, data indicated that all 
observers were able to independently 
observe common characteristics of 
effective teaching. Agreement across 
observers, who observed independent 
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Table 1 . Comparison of Observation Feedback for Fourth Observation for Teacher 1 by University Faculty 


Teacher 1 




Evaluator 



Lateral-Entry Teacher 
Middle School Math 

University Supervisor A 

University Supervisor B 

Director of Field 

Experiences 



Frequency of Occurrence in Observation Feedback Form 


Emergent Observation 
Feedback Themes 

Identified 

Strength 

Recommend 

Improvement 

Identified 

Strength 

Recommend 

Improvement 

Identified 

Strength 

Recommend 

Improvement 

Instructional Strategies 

2 

1 

2 

1 

1 

1 

Classroom Management 

2 

1 

3 

1 

2 

1 

Organization 

2 


3 


2 


Questioning Techniques 

2 


2 


1 


Student Engagement 

3 

1 

4 

1 

2 

1 

Content 

3 


2 


2 


Feedback/ Assessment 

2 

1 

3 

1 

2 

1 


Table 2. Comparison of Observation Feedback for Fourth Observation of Teacher 2 by University Faculty 


Teacher 2 


Evaluator 


Student Teacher 


Director of Field 


Secondary Social Studies 

Emergent Observation 
Feedback Themes 

University Supervisor A University Supervisor B Experiences 

Frequency of Occurrence in Observation Feedback Form 

Identified 

Strength 

Recommend 

Improvement 

Identified 

Strength 

Recommend 

Improvement 

Identified 

Strength 

Recommend 

Improvement 

Instructional Strategies 

2 

1 

2 

1 

2 

1 

Classroom Management 

2 

1 

2 

1 

1 

1 

Organization 

1 

1 

1 

2 

1 

2 

Questioning Techniques 

2 


2 


2 


Student Engagement 

2 

1 

2 

1 

1 

1 

Content 

1 

1 

3 

3 

1 

1 

Feedback/Assessment 


1 

1 

1 

1 

1 


of each other, affirmed that the remote 
observation process did not impede 
their ability to evaluate each interns 
teaching performance. 

Are There Differences in the Remote and 
Face-To-Face Observations? If So, What 
Are These Differences? 

The analysis of the observational tran¬ 
scripts unveiled five emergent themes 
regarding the differences in the observa¬ 
tional processes. These data present the 
best description for how all participants 
perceived these differences. Themes 
included sound, video, curriculum 
materials, observation forms, and added 
value of technology. 

Sound. The data revealed problems 
and challenges related to sound as well 
as success and added value. During the 


first observation of the first intern, there 
was no sound. The camera did not have 
an external microphone, so the observ¬ 
ers were not able to hear the graduate 
intern and were able only to view the 
graduate intern. This issue was resolved 
by using an external microphone con¬ 
nected to the computer. During the first 
observation of the second intern, the 
observer was able to use the external 
microphone; however, what was heard 
was limited. The students comments 
were not clear, and when the graduate 
interns posed questions, the observers 
could not hear the students responses. 
Small-group conversations were not 
audible. The supervisors had to request 
clarifications from the camera operator. 
At times, even the teacher was difficult 
to hear. The external microphone was 


effective only with teacher-directed in¬ 
struction, and even then it was limited. 
This experience led to the use of a wire¬ 
less microphone. 

With the implementation of the 
wireless microphone, sound quality 
drastically improved. Not only could the 
teacher could be heard clearly, but this 
device provided access to teacher-stu¬ 
dent individual interactions that were 
not accessible in face-to-face observa¬ 
tions. The observers could hear whispers 
between teacher and student, making it 
possible for observers to access teacher 
feedback. Evaluations included nuisanc¬ 
es of teaching, such as student behavior 
and discipline actions, as well as teacher 
feedback and one-to-one instruction. 
Teacher-student conversations were not 
observable in face-to-face observations, 
so the remote setting uncovered a new 
layer of teaching evaluations. 

Video. The data analysis revealed 
limitations of the streaming video. The 
video in Centra provided a limited 
scope of what is viewable in the class¬ 
room. All students were not visible on 
the screen at one time, as the supervisor 
could see only the view that the cam¬ 
era projected. There is a need to scan 
the room to see the intricate aspects 
of teaching, such as what all students 
are doing and where the teacher is in 
proximity to students with discipline is¬ 
sues or students with special needs. The 
video does not provide personalization. 
The students facial expressions are dif¬ 
ficult to read unless the image is close. 
The supervisor can see only from one 
camera angle, which may not capture 
both the student’s and teachers expres¬ 
sions in detail. The size of the video 
is also limited, which is a limitation 
of Centra. Only two sizes of viewing 
screen are available through Centra. 
When using the larger undocked view, 
the images become somewhat distorted 
because pixilation is reduced. The clarity 
of the video is also restricted. This again 
is due to the limitations of Centra and of 
the camera. Finally, the room and light¬ 
ing affects the observation. The windows 
can lighten or darken the video, so room 
lighting is essential. Both of these can 
disturb the clarity of visual images. 
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Table 3. Round-Trip Cost for One Observation at Each School 


School 

Mileage 

Cost* 

Travel Time 

Middle 

72.72 

$42.54 

1 hour 22 minutes 

High 

75.1 

$43.92 

1 hour 20 minutes 

Total 

147.82 

$86.46 

2 hours 44 minutes 


Curriculum materials. In a tradi¬ 
tional face-to-face observation, the 
graduate intern provides all curriculum 
materials for the university supervi¬ 
sor to view both before and during the 
observation. During the initial obser¬ 
vation, we realized the importance of 
access to these materials. We asked the 
graduate interns to e-mail instructional 
materials to the supervisor prior to 
the observation. This provided several 
benefits for the supervisor. The supervi¬ 
sor uploaded PowerPoint presentations, 
lesson resources, readings, political 
cartoons, etc., to Centra for all observ¬ 
ers to view simultaneously with teach¬ 
ing observations. The observer fol¬ 
lowed curriculum materials during the 
teaching of the lesson because Centra 
allowed two windows to be viewable at 
the same time. Centra document-shar¬ 
ing features also proved to be beneficial 
during the postconference with the 
graduate intern. Observers provided 
feedback immediately so the intern 
could read instructor comments during 
the debriefing session. 

Observation forms. During the initial 
observation for each intern, the supervi¬ 
sors did not have access to electronic 
versions of the observation forms. This 
served as a barrier during the postcon¬ 
ference, as the supervisors could not 
share the forms with the interns, as they 
could in face-to-face observations. We 
obtained e-versions of these forms and 
used them for subsequent observations. 
This was a benefit in that the observation 
forms and evaluations could be shared 
with the graduate interns via Centra 
during the postconference. 

Added value of technology. Although 
we uncovered several limitations dur¬ 
ing these remote observations, we also 
discovered several benefits. Multiple 
observers could take part in an obser¬ 
vation. This provided different lenses 
on learning, from both the pedagogical 
and content-area sides. ROGI allowed 
for limited class intrusion. Multiple 
observers in a classroom would cause a 
disruption in a traditional face-to-face 
observation and affect student behavior 
as well as intimidate novice teachers. 
ROGI “encouraged real teaching and 


*Reimbursement per mile ($.0585) in this state 

not just a performance.” The camera 
size was small and less noticeable to 
both the graduate interns and students 
than a person sitting in the room 
observing. Interns noted forgetting 
that the camera was in the classroom; 
however, in face-to-face observations, 
they did not indicate these feelings and 
instead acknowledged differences in 
student behavior from the norm as well 
as in their own nervousness about be¬ 
ing observed. 

Centra allowed uploading written 
feedback that was both viewable during 
the postconference and available for 
multiple viewers. This provided im¬ 
mediate, rich feedback on the lesson 
plan, observation, and curriculum 
materials. This immediate feedback 
simulated that of a traditional face-to- 
face postconference. The graduate intern 
received recommendations for instant 
implementation. Interns noted in their 
final interviews that they made changes 
to subsequent lessons the day of the 
observation. 

Cost-benefit analysis revealed 
significant cost differences in modes of 
observation. A single university supervi¬ 
sor conducting a total of five face-to-face 
observations for two graduate interns 
would have cost the university a total of 
$864.60. For two university supervisors 
to provide the same number of observa¬ 
tions, as occurred in this study, would 
have cost $1,723.20. The general practice 
is for graduate interns to be assigned a 
single university supervisor, so the latter 
figure is not truly reflective of actual 
costs but does present the argument 
that ROGI offers new alternatives to the 
observational process than are possible 
in traditional settings. These figures, in 
comparison to the two observations that 
were conducted face to face for each 
intern, was significantly higher than the 
$345.84 that was incurred during the 
pilot study. 


Another added value of ROGI was 
travel time saved (see Table 3). Although 
observation time did not vary depend¬ 
ing on mode of observation, travel time 
to these remote face-to-face sites does. 

It would have taken two university 
supervisors 54 hours and 40 minutes for 
roundtrip travel to conduct five obser¬ 
vations for two graduate interns. The 
actual time savings for the university 
supervisors in this study were 32 hours 
and 48 minutes. Time savings in travel 
could potentially affect university super¬ 
visors’ workload capacity by enabling 
them to observe more interns during a 
semester for the same amount of time as 
face-to-face observations. 

Do Differences in Observationai 
Processes Affect the Quaiity of 
Teacher Preparation? 

Additional qualitative data from sum- 
mative interviews provided evidence 
that face-to-face and remote observa¬ 
tions were interchangeable processes, 
based on the comparison of experiences 
of interns and university supervisors. 

In both settings, university supervisors 
observed each intern teach a lesson 
and then conducted a postconference 
immediately following observation. For 
remote observations, they conducted 
the postconferences using a webcam, 
headset with microphone, and the ROGI 
classroom technology. Both interns 
commented that they felt comfortable 
with classroom observations and noted 
personal and professional growth as a 
result of the overall internship experi¬ 
ence. Teacher 1 commented that she 
found observations helpful and stated 
that “to see and to give you some input 
on knowledge that they may have, things 
that may help you with things that went 
wrong with that one class.... That is very 
helpful.” Teacher 2 acknowledged that 
he “enjoyed the observations. I felt that 
professors actually seeing ... how you are 
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managing your classroom in approach¬ 
ing your lesson. Its very beneficial to 
your teaching futureThe interns com¬ 
mented on the importance of having 
an expert observe their teaching and 
the value of feedback in helping them 
recognize strategies for improvement. 
They noted gaining positive feedback 
and strategies for improvement and did 
not distinguish differences in teach¬ 
ing evaluations based on the mode of 
observation. Teacher 1 commented that 
she valued getting “a different per¬ 
spective from someone observing my 
class.... Everything they told me about 
in the debriefing I found helpful. I tried 
to use these suggestions and be better 
or improved on them the next time they 
observed me.” Teacher 2 affirmed these 
feelings in his interview: “They [univer¬ 
sity supervisors] can see firsthand how 
you are doing things, and they [univer¬ 
sity supervisors] can comment on the 
spot.... They observe you and correct 
you on the spot. I felt that face-to-face 
and remote observations can do the 
same thing. They can fix it right there.” 

In addressing a question asking interns 
to compare the observation processes, 
Teacher 1 stated that the observation 
process was “not that much different 
in that you still get to the conference 
afterwards, and the technology was 
wonderful.” She did, however, frequently 
(in 13 separate comments) note in her 
interview that she was more comfortable 
during the remote observation process 
than face-to-face. She indicated that that 
technology enabled a more authentic 
snapshot of her teaching. Teacher 2 
stated that “both observation processes 
helped me. Yeah, I felt both of them 
[face-to-face and remote observations] 
were beneficial... to my success.... 

There is no, one or the other that was ... 
that weighed more heavily on me being 
successful or unsuccessful.” Although he 
did equate the processes, he was not as 
comfortable with the remote observa¬ 
tion because he not did have someone in 
the classroom in case things went awry 
with his teaching. 

University supervisors also spoke 
about their experiences as observers in 
both face-to-face and remote settings 


in their interviews. All supervisors felt 
comfortable using either observation 
processes and noted that they felt that 
they could assess interns’ teaching 
effectively assessed in either setting. 
University Supervisor A commented, 

“I think we have tried to simulate 
the face to face so much that we have 
found the remote observation to be 
reliable, reliable between supervi¬ 
sors, and from experience I do think 
they are both effective.” University 
Supervisor B stated that “... overall 
both methods of observation are very 
comparable.” The director of field ex¬ 
periences also affirmed these data by 
stating that she did not “feel that there 
were major differences in watching an 
intern teach on the computer versus 
sitting in a classroom.” Based on these 
and aforementioned analyses, assess¬ 
ment data provide evidence that the 
remote observations are interchange¬ 
able with face-to-face observations in 
measuring the overall effectiveness of 
teaching. 

Discussion and Implications 

In comparing the two observational 
processes of face-to-face and remote 
(ROGI), we found that the modes of 
observation are not equivalent methods 
for evaluating graduate interns; however, 
data suggest that these processes are 
comparable. Each mode of observation 
has both benefits and limitations, but 
neither process was overall a more effec¬ 
tive method of evaluating the quality of 
teaching. 

Equivalency means that the observer 
and intern had the exact same experi¬ 
ence despite the mode of observation, 
which did not occur in this study. Each 
mode of observation presented differ¬ 
ences in observational experiences for 
participants, although these were inter¬ 
preted as both benefits and limitations. 
Although face-to-face observations al¬ 
lowed the observer more autonomy over 
what was observable as well as clear, 
discernable views of facial expressions, 
the process was more intrusive to the 
graduate interns and presented a staged 
lesson that was not truly reflective of 
teaching practices and student behavior. 
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ROGI, in contrast, was less intrusive 
and presented a more authentic view of 
teaching and learning; however, it did 
not allow for full disclosure of the entire 
classroom at all times, and facial expres¬ 
sions were less vivid. One significant 
contrast in the mode of observation was 
what the observers heard. Face-to-face 
observations provided an environment 
in which student voice and participation 
were easily understood; however, when 
students were working in small groups 
or independently with the teacher, stu¬ 
dent voices were not distinguishable. In 
contrast, ROGI with the use of a wireless 
microphone provided opportunities to 
hear teacher-student and student-stu¬ 
dent exchanges in both small groups 
and individual interactions; yet, in the 
whole-class setting, student input was 
difficult to understand. Both of these 
examples as well as overall data findings 
suggest that neither mode of instruction 
was without flaw. Despite limitations 
posed by the modes of observation, 
benefits were gained in both processes, 
distinguishing each as interchangeable 
but not equivalent. 

By comparison, we purport that 
observational experiences for all partici¬ 
pants were similar and interchangeable 
without compromising the quality or 
value of the internship in promoting 
professional growth. As these data in¬ 
dicate, the mode of observation did not 
inhibit the observer from deciphering 
elements of effective teaching and good 
teaching behaviors; thus, the observer, 
whether sitting in the classroom or 
viewing synchronously, was able to 
provide essential feedback to help the 
graduate interns improve their peda¬ 
gogical practices. Data analyses suggest 
that the mode of observation did not 
affect the overall outcome of improve¬ 
ment for both graduate interns. Feed¬ 
back from observations, whether they 
were conducted face to face or remotely, 
identified common elements of success¬ 
ful teaching. Differences in summative 
evaluation of the interns’ performance 
were attributed to differences between 
the two interns and observer expertise. 

Data also indicated that ROGI does of¬ 
fer a viable and fiscally wise alternative for 
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conducting teaching observations. Over 
time, the investment in hardware and 
software was more cost effective and time 
efficient than would have been possible 
if five face-to-face observations had been 
required for each intern. Additionally, 
having a second university supervisor is 
both cost and time prohibitive; however, 
ROGI allowed for additional expertise 
and cross-validation of performance 
assessment of graduate interns. Thus, 
we conclude that, as both observation 
processes are comparable and effective 
teaching can be measured either face to 
face or remotely, the added layers of the 
evaluative process and the potential cost 
benefit of technology-mediated observa¬ 
tion is a justifiable alternative, challeng¬ 
ing traditional perceptions of teacher 
preparation. 

As colleges and universities seek 
innovative strategies to address teacher 
shortages, increase enrollment by tap¬ 
ping new populations, and respond to 
economic pressures for lower operating 
costs, ROGI offers an opportunity to 
expand educational outreach for teacher 
licensure programs beyond traditional 
geographical boundaries. This research 
provides possible uses of technology 
that can be expanded to address growing 
pedagogical needs in an evolving online 
teaching and learning community. To 
fully embrace the capabilities of emer¬ 
gent technologies, additional research is 
needed to explore the effectiveness and 
feasibility as well as the impact of the 
widespread use of ROGI. 
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